Skip to content

feat(spanner): add built-in tracing to Cloud Trace - #15064

Draft
uditdasari wants to merge 3 commits into
mainfrom
spanner-builtin-tracing
Draft

uditdasari wants to merge 3 commits into
mainfrom
spanner-builtin-tracing

Conversation

@uditdasari

@uditdasari uditdasari commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

This adds built-in tracing. Setting SPANNER_ENABLE_TRACING=true (or SpannerOptions.Builder#setEnableBuiltInTracing(true)) makes the client export its traces to Cloud Trace and turns on end to end tracing, without any OpenTelemetry setup in the application. It is off by default, because Cloud Trace charges for the spans that are exported.

What it does

  • When built-in tracing is on and the application has not configured OpenTelemetry tracing, the client creates its own OpenTelemetry instance with an OTLP gRPC exporter, a batch span processor, a parent based trace id ratio sampler and the W3C trace context propagator. It is never registered as the global OpenTelemetry, so tracing of the rest of the application does not change.
  • The spans are exported to the Cloud Telemetry (OTLP) API at telemetry.googleapis.com, which Cloud Trace recommends instead of the Cloud Trace exporter. The exporter uses the credentials of the client, and the spans carry the gcp.project_id resource attribute with the project of the client. The service name is taken from OTEL_SERVICE_NAME if that is set.
  • The instance is shared by all clients of the same project and sampling rate in the process. Pending spans are exported when the JVM shuts down.
  • End to end tracing is turned on as well, so traceparent and x-goog-spanner-end-to-end-tracing are sent and Spanner adds its server side spans to the same trace.
  • The sampling rate is 0.1 by default. It can be set with the standard OTEL_TRACES_SAMPLER_ARG environment variable or with setBuiltInTracingSamplingRate(double). The startup log of the client shows whether built-in tracing is enabled, and the sampling rate when it is enabled.
  • If no tracing framework has been chosen yet, built-in tracing chooses OpenTelemetry. This is also done for clients that are created by the Connection API (JDBC, PGAdapter), because ConnectionSpannerOptions.Builder#build() does not call SpannerOptions.Builder#build().
  • The Spanner spans, the Connection API spans, the traceparent header and API tracing (if enabled separately) use the built-in instance. Metrics are not affected.

When it is not used

  • The OpenTelemetry instance that the client would otherwise use has an enabled tracer (Tracer#isEnabled()), for example an OpenTelemetry set in SpannerOptions with a tracer provider that exports spans, or a global OpenTelemetry registered by the application or by the OpenTelemetry Java agent. That configuration is used instead, end to end tracing is still turned on, and a log line says which instance is used. An OpenTelemetry instance that is only used for metrics does not disable built-in tracing.
  • The application uses OpenCensus tracing. Built-in tracing is not used, end to end tracing is not changed, and a warning is logged.
  • The client uses no credentials, the emulator, Spanner Omni, or a universe domain other than googleapis.com. The Telemetry API and Cloud Trace are not available in other universes.

Requirements

The project of the client needs the Telemetry API and the Cloud Trace API enabled, and the credentials need the telemetry.traces.write permission, which is included in roles/cloudtrace.agent, roles/telemetry.tracesWriter and roles/telemetry.writer. User credentials also need a quota project.

Behavior for existing applications

  • Nothing changes when built-in tracing is off, apart from one extra line in the startup log.
  • If an application turns on built-in tracing and then calls the deprecated SpannerOptions.enableOpenCensusTraces() after it has created a client or a connection, that call now throws, because the tracing framework is already set to OpenTelemetry. Calling it before creating any client still works, and then built-in tracing is not used.

Dependencies

io.opentelemetry:opentelemetry-sdk-trace, io.opentelemetry:opentelemetry-exporter-otlp and io.opentelemetry:opentelemetry-exporter-sender-grpc-managed-channel are now compile dependencies. Their versions are managed by the OpenTelemetry BOM in the shared dependencies. The OkHttp sender of the OTLP exporter is excluded, so the exporter uses gRPC. These dependencies are added for all applications, also when built-in tracing is off, and can conflict with other versions of these libraries that an application already uses. This should be called out in the release notes.

Testing

  • BuiltInTracingTest runs a client against a mock server and checks that the request carries a sampled traceparent and the end to end tracing header, that the exported spans have the same trace id and the gcp.project_id resource attribute, that clients of the same project share one instance and other projects get their own, that an OpenTelemetry instance with tracing is used instead of built-in tracing, that a metrics only OpenTelemetry instance (set in the options or global) does not disable built-in tracing, and that built-in tracing is not used without credentials, in another universe domain, or with OpenCensus. It also checks the request headers and the service name.
  • BuiltInTracingConnectionTest checks that the Connection API spans and the Spanner spans are exported by built-in tracing.
  • Tests that mock SpannerOptions now also stub getTracingOpenTelemetry().
  • Manually: a span exported by the built-in exporter with user credentials arrived in Cloud Trace through the Telemetry API. With the OpenTelemetry Java agent 2.33.1, the agent's OpenTelemetry instance is used and built-in tracing is not, and with OTEL_SDK_DISABLED=true built-in tracing is used.

Setting SPANNER_ENABLE_TRACING=true (or SpannerOptions.Builder#setEnableBuiltInTracing(true))
makes the client export its traces to Cloud Trace and turns on end to end tracing, without any
OpenTelemetry setup in the application. It is off by default.

This adds com.google.cloud.opentelemetry:exporter-trace and io.opentelemetry:opentelemetry-sdk-trace
as compile dependencies for all applications, also when built-in tracing is off. They can conflict
with other versions of these libraries that an application already uses.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces built-in tracing capabilities to the Google Cloud Spanner Java client using OpenTelemetry. It adds configuration options to SpannerOptions to enable built-in tracing and set a sampling rate, either programmatically or via environment variables. A new BuiltInTracingProvider class is introduced to manage and cache OpenTelemetry instances per project and sampling rate. The Spanner client implementation and RPC layers are updated to use this new tracing mechanism. Feedback on the changes suggests wrapping the Runtime.getRuntime().addShutdownHook call in BuiltInTracingProvider within a try-catch block to handle potential IllegalStateExceptions if the JVM is already shutting down.

Comment on lines +77 to +80
if (shutdownHook == null) {
shutdownHook = new Thread(this::shutdown);
Runtime.getRuntime().addShutdownHook(shutdownHook);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Registering a shutdown hook via Runtime.getRuntime().addShutdownHook can throw an IllegalStateException if the JVM is already in the process of shutting down. Since this call is outside the try-catch block, it can cause unexpected crashes for callers initializing or using a client during JVM shutdown. Wrapping the registration in a try-catch block for IllegalStateException resolves this safely.

      if (shutdownHook == null) {
        Thread hook = new Thread(this::shutdown);
        try {
          Runtime.getRuntime().addShutdownHook(hook);
          shutdownHook = hook;
        } catch (IllegalStateException e) {
          // JVM is shutting down, ignore
        }
      }

Use exporter-trace 0.33.0 so that its dependencies match the versions in the shared dependencies.
Do not use built-in tracing when the client uses a universe domain other than googleapis.com.
Only log the sampling rate when built-in tracing is enabled, and log which OpenTelemetry instance
is used when built-in tracing is not used.
Built-in tracing now exports spans with the OpenTelemetry OTLP gRPC exporter to the Cloud Telemetry
(OTLP) API instead of the Cloud Trace exporter, as recommended by Cloud Trace. The OpenTelemetry
artifacts are already part of the shared dependencies, so no new third-party dependency is added.
Spans carry the gcp.project_id resource attribute and the service name from OTEL_SERVICE_NAME.

Built-in tracing is now also used when the configured OpenTelemetry instance has no enabled
tracer, for example when an application uses OpenTelemetry only for metrics.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant